Image processing apparatus

ABSTRACT

An image processing apparatus in which a plurality of images obtained by photographing a three-dimensional object from a plurality of viewpoints are integrated by using image-pickup parameters such as position, pose, focal lengths, and aberration information of a camera at these viewpoints, thereby integrating shapes at arbitrary viewpoints of the three-dimensional object.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image measuring method for usingimages from a plurality of viewpoints to simply and accurately extractimage-pickup parameters in photographing an object, e.g., the pose,position, base length, vergence angle, focal length, and aberrationinformation of a camera, and an image processing apparatus for measuringthe three-dimensional shape of an object on the basis of image-pickupparameters to display the three-dimensional shape.

2. Description of the Related Art

A conventional method of measuring three-dimensional shape data of anobject from an image, using a stereo measuring method by a binocularcamera is known, as described in the “Image Processing Handbook” editedby Onoue, et al, Shokodo, 1987. A method of extracting relativepositions (parallel movement amount and rotational movement amount) ofall shape data of the object and a method of integrating the relativepositions, which methods are based on the above stereo measuring method,are disclosed in Japanese Unexamined Patent Publication Nos. 5-303629and 6-208610, the Institute of Electronics Information and CommunicationEngineers paper D-II (Vol. J75-D-II, pp. 737-748, 1992).

According to these methods, the relative positions of a plurality ofpartial shape data are calculated by using only the image of an objectso as to calculate a whole three-dimensional shape without directlymeasuring the relative positions of the object and a camera bysequentially connecting the partial shape data to the camera. A methodin which the camera has a pose detection sensor arranged thereon to usepose data and image data is disclosed in Japanese Unexamined PatentPublication No. 6-241731. In addition, a method of calibrating thethree-dimensional position and pose of an image-pickup means by imagemeasurement is disclosed in Japanese Unexamined Patent Publication No.6-259536.

However, of the above prior art, the method disclosed in JapaneseUnexamined Patent Publication Nos. 5-303629 and 6-208610, the Instituteof Electronics Information and Communication Engineers paper D-II (Vol.J75-D-II, pp. 737-748, 1992) cannot always accurately calculate therelative positions between the camera position and the partial shapeswithout measuring the shape of the object and a pattern on the objectsurface, and the following problems arise. That is, errors occurring inintegrating shape data are accumulated, and a calculation amount becomeslarge.

According to the method disclosed in Japanese Unexamined PatentPublication No. 6-241731, since parameters other than the pose of thecamera are obtained on the basis of the image data, the same problems asdescribed above arise. In any example, the base length, vergence angle,focal length, distortion, and aberration information of the camera mustbe accurately calibrated in advance. Therefore, when an object isphotographed, the position and pose of the camera, which parameters areinherent in the camera cannot be adjusted.

In Japanese Unexamined Patent Publication No. 6-259536, the followingproblems arises. That is, accuracy and calculation stability are poorbecause an image at one viewpoint positionis used, or the same problemsas described in the prior art arise when the method is applied tothree-dimensional shape measurement because the focal length of theimage-pickup means is not extracted.

SUMMARY OF THE INVENTION

The present invention has been made to solve the above problems of theprior art, and has as its object to provide an image measuring methodand apparatus capable of easily extracting an image-pickup parameter ata high accuracy by inputting an object image, and measuring the image.

Therefore, according to a preferred embodiment of the present invention,there is disclosed an image processing method and apparatuscharacterized in that, on the basis of images from a plurality ofviewpoint positions including a known pattern and an object,image-pickup parameters of the image are extracted.

According to another preferred embodiment of the present invention,thereis disclosed an image measuring method and apparatus characterizedinthat a pattern having features whose positional relationship is knownis defined as a first object, and a predetermined object is defined as asecond object, and image-pickup parameters are extracted on the basis ofimages from a plurality of viewpoint positions of the two objects.

It is another object of the present invention to measure athree-dimensional shape at a high accuracy with a simple arrangement.

In order to achieve the above objects, according to still anotherpreferred embodiment of the present invention, there is disclosed animage measuring method and apparatus characterized by input means forinputting unknown object images obtained by viewing, at a plurality ofviewpoint positions, a first object having features whose positionalrelationship is known and a second object having a three-dimensionalshape which is at least partially unknown; image-pickup parameterextracting means for extracting image-pickup parameters corresponding tothe object images; three-dimensional shape information extracting meansfor extracting three-dimensional shape information of the second object;recording means for recording the three-dimensional shape information;and image display means for displaying an image.

According to still another preferred embodiment of the presentinvention, there is disclosed an image measuring method and apparatuscharacterized by image inputting means for inputting unknown objectimages obtained by viewing, at a plurality of viewpoint positions, afirst object having features whose positional relationship is known anda predetermined second object; means for extracting image-pickupparameters corresponding to the object images; image recording means forrecording the object images; image-pickup parameter recording means forrecording parameters corresponding to the object images; and image.display means for displaying an image.

The other objects and characteristic features of the present inventionare understood with reference to the following specification anddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a view showing the arrangement of Embodiment 1, and FIG. 1Bis a view showing the arrangement of an image-pickup means.

FIGS. 2A and 2B are views for describing known patterns used in thisembodiment.

FIGS. 3A and 3B are views for describing known patterns used in thisembodiment.

FIG. 4 is a flow chart of an entire process of this embodiment.

FIG. 5 is a flow chart of an image-pickup parameter extraction process.

FIG. 6 is a view for describing a recording format of an image-pickupparameter corresponding to image data.

FIG. 7 is a view for describing a recording format of an image-pickupparameter corresponding to image data.

FIG. 8 is a flow chart of a three-dimensional shape data extractionprocess.

FIG. 9 is a flow chart of a partial shape data integration process.

FIG. 10 is a flow chart of an image-pickup parameter extraction processaccording to Embodiment 2.

FIGS. 11A and 11B are views for describing image-pickup parameterextracting objects with known patterns having known shapes used inEmbodiment 3.

FIG. 12 is a view showing the arrangement of a pattern and animage-pickup means when a known (semi) transparent pattern is used inEmbodiment 4.

FIG. 13 is a flow chart of an entire process when the position and poseof an image-pickup means are extracted by using a known pattern inEmbodiment 5 to be used in image reproduction.

FIG. 14 is a view showing the arrangement of a system portion when anauxiliary projection means is used in Embodiment 6.

FIG. 15 is a view for describing a camera coordinate system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In order to achieve the above objects, according to an embodiment of thepresent invention, an image measuring method is characterized in that,on the basis of images from a plurality of viewpoint positions includinga known pattern and an object, image-pickup parameters of the images areextracted. For this reason, required parameters can be directly measuredat a high accuracy from the image without recording image-pickupparameters including the position and pose of an image-pickup means inphotographing the object.

According to another embodiment of the present invention, an imagemeasuring method is characterized in that a pattern having featureswhose positional relationship is known is defined as a first object, anda predetermined object is defined as a second object, and image-pickupparameters are extracted on the basis of images from a plurality ofviewpoint positions of the two objects. For this reason, the positionalrelationship between an image-pickup means and the first and secondobjects and the image-pickup parameters in picking up an image aremeasured on the basis of only the image.

According to still another embodiment of the present invention, an imagemeasuring method is characterized in that the known pattern is presenton an object whose shape is at least partially known. For this reason,even if another object having a known pattern is always prepared,image-pickup parameters are measured at a high accuracy.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized by: input means for inputtingunknown object images obtained by viewing, at a plurality of viewpointpositions, a first object having features whose positional relationshipis known and a second object having a three-dimensional shape which isat least partially unknown; image-pickup parameter extracting means forextracting image-pickup parameters corresponding to the object images;three-dimensional shape information extracting means for extractingthree-dimensional shape information of the second object; recordingmeans for recording the three-dimensional shape information; and imagedisplay means. For this reason, even if the image-pickup parameters arechanged by the viewpoint positions, types of objects, illuminationconditions, and the like, estimation of parameters, especially,estimation of the pose and position of an image-pickup means, areperformed at a high accuracy on the basis of the input images, andsimple and accurate three-dimensional information is extracted.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized in that input means is constitutedby image-pickup means and storage means for storing an image from theimage-pickup means. For this reason, when only object images from aplurality of viewpoint positions including a known pattern are picked upin advance, extraction of image-pickup parameters and extraction ofthree-dimensional shape information is performed as post-process.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized in that three-dimensional shapeextracting means is constituted by reference coordinate system settingmeans having a predetermined point on the first object as an origin anda three-dimensional shape extracting means for the second object in thereference coordinate system. For this reason, a plurality of partialshapes obtained by photographing the same object at different anglesaplurality of times is integrated at a high accuracy.

According to still another embodiment of the present invention, an imagemeasuring method is characterized in that the image-pickup parametersinclude at least one of the position, pose, focal length, and aberrationinformation of the image-pickup means. For this reason, extraction ofhighly accurate three-dimensional shape information and generation of anobject image from an arbitrary viewpoint position is easily performed.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized in that the image-pickup meansincludes a plurality of optical path means, at least one imaging means,and photoelectric conversion means. For this reason, image pickupconditions of image input for extracting three-dimensional shape data isstably controlled.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized in that the image-pickup means is abinocular camera having at least one of base length adjusting means,vergence angle adjusting means, and focal length adjusting means. Forthis reason, a known pattern and an object are photographed underoptimum image pickup conditions, so that three-dimensional shapesobtained at different viewpoint positions are integrated at a highaccuracy.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized by: input means for inputtingunknown object images obtained by viewing, at a plurality of viewpointpositions, a firstobject having features whose positional relationshipis known and a predetermined second object; image-pickup parameterextracting means for extracting image-pickup parameters corresponding tothe object images; two-dimensional image recording means for the secondobject; means for recording image-pickup parameters corresponding to thetwo-dimensional image of the second object; and image display means. Forthis reason, simple and smooth reproduction or synthesization of anobject image viewed from an arbitrary viewpoint position is performed byusing only input image data.

According to still another embodiment of the present invention, an imagemeasuring method is characterized in that the known pattern isconstituted by a plurality of patterns respectively havingtwo-dimensional features which are different from each other. For thisreason, the position of a feature point is calculated on the basis of animage, and a process of extracting stable and high-accurate image-pickupparameters and a three-dimensional shape data is performed.

According to still another embodiment of the present. invention, animage measuring apparatus is characterized in that the image-pickupmeans has two eyes whose fixed focal lengths are different from eachother, one eye is used to acquire the first object image, and the othereye is used to acquire the second object image. For this reason, anerror of image-pickup parameters or three-dimensional shape data causedby one blurred image obtained when an unknown-shape object and a knownpattern are photographed within the same field of view is suppressed.

According to still another embodiment of the present invention, an imagemeasuring apparatus is characterized in that the first object isconstituted by arranging light-emitting elements. For this reason,image-pickup parameters and three-dimensional shape data are extractedat a higher accuracy.

Embodiments of the present invention are described in detail below withreference to the accompanying drawings.

Embodiment 1

FIG. 1A is a view showing the basic arrangement of this embodiment.Reference numeral 1 denotes a binocular camera which functions as animage-pickup means; 2, a planar object having a known pattern; 3, anobject which is placed on the planar object 2 and whosethree-dimensional shape is to be measured; 4, an image processing means;5, a monitor for an image from the binocular camera 1; and 12, a shapedata storage means.

The image-pickup means 1, as shown in FIG. 1B, basically has left andright cameras 1 _(L) and 1 _(R), an optical axis direction adjustingmeans. (vergence angle adjusting means) 6 for the left and right cameras1 _(L) and ¹ _(R), a base length (e.g., distance between the main pointpositions of the imaging systems of the left and right cameras 1 _(L)and 1 _(R)) adjusting means 7, photoelectric transforming means 8 _(L)and 8 _(R), focal length adjusting means 9 _(L) and 9 _(R), a maincontroller 10, and an image storage unit 11 having an A/D converterfunction. The image-pickup means may have a known blur correctingmechanism (not shown) for correcting the flow and blur of an image byunstably handling the image-pickup means, to stabilize the image.

A measurer holds the binocular camera 1 and photographs the object 3 ata plurality of positions while checking the images of the planar object2 and the three-dimensional shape object 3 with the monitor 5. Note thatthe image processing means 4 may be built in the image-pickup means 1.

FIGS. 2A and 2B and FIGS. 3A and 3B show patterns on the planar object 2as used in this embodiment. In the pattern in FIG. 2A, feature pointsP₁, P_(2, P) ₃, . . . are colored dot patterns respectively havingdifferent hues, and are arranged at predetermined distribution densityto form a plurality of concentric circles. As a modification of thepattern of this type, a pattern in which dots are arranged to formellipses which have the same center and different long-axis sizes anddifferent short-axis sizes may be used. In FIG. 2A, the sizes of dotschange in the radial direction, the dots on the same cycle have the samebrightness (but different hues). This pattern is not limited to theabove arrangement, and the sizes, brightness, and hues of dots may bearbitrarily determined. For this reason, it is satisfactory that thedots have the different attributes, respectively. In FIG. 2B, a coloreddot pattern similar to that in FIG. 2A is arranged at a predetermineddistribution density on rectangles having the same centroid anddifferent sizes. As a modification of the pattern of this type, apattern in which dots are arranged on polygons having the same centroidand different sizes may be used.

FIG. 3A shows a grating pattern having a predetermined pitch and apredetermined cross angle. In this pattern, at least one of thebrightnesses, hues, and saturations of respective lines are differentfrom each other. In FIG. 3A, each crossing point serves as a featurepoint. FIG. 3B shows a pattern in which X-shaped cross patterns havingdifferent hues are arranged on the grating similar to the gratingpattern in FIG. 3A. In addition, two-dimensional pattern elements (e.g.,L-shaped corner elements having different directions and different crossangles, characters, or the like) may be arranged at a predetermineddistribution density on a grating, rectangles, concentric circles, orthe like. The above patterns may be constituted such that light-emittingelements are arranged in the patterns.

As described above, as a known pattern, a pattern having features whosepositional relationship (relative coordinate) is known and which havedifferent colors such that the features are easily identified on animage, or a pattern including different two-dimensional pattern elementsis especially preferable. A reference orthogonal coordinate system isdefined as follows. That is, for example, two adjacent feature pointsobserved when the first image including the images of the objects 2 and3 is picked up by using the binocular camera 1 are selected, a straightline extending from one (origin) of the two feature points to the otheris defined as an X axis, a Y axis is set in the plane of the planarobject, and a Z axis is set to be perpendicular to the X and Y axes andto be a right-hand system. It is assumed that the positions of thefeature points P₁, P₂, . . . are represented by (X₁, Y₁, 0), (X₂, Y₂,0), . . . .

Image-pickup parameters are described below. The focal lengths of theright and left cameras are represented by f_(L) and f_(R), respectively,a base length is represented by B, and Euler angles expressing therelative pose of the right and left cameras are represented by α₀, β₀,and γ₀, respectively. The horizontal and vertical direction coordinatesof an in-image optical axis center position of the left camera arerepresented by u₀ ^(L) and v₀ ^(L), respectively, and the horizontal andvertical direction coordinates of the right camera are represented by u₀^(R) and u₀ ^(R), respectively. FIG. 15 is a view for explaining a leftcamera coordinate system (X_(L), Y_(L), Z_(L)). An origin O_(L) is amain point in a perspective coordinate system, (x_(L), y_(L)) is anorthogonal coordinate system on a sensor plane, and (u, v) is an imageframe coordinate system on the coordinate system (x_(L), y_(L)). Theposition of an origin O of image frame coordinate system is given by (u₀^(L), v₀ ^(L)). The same parameters as described above are set in theright camera coordinate system (X_(R), Y_(R), Z_(R)). Coordinateconversion from a reference coordinate system P=(X, Y, Z)^(T) to theleft camera coordinate system C_(L)=(X_(L), Y_(L), Z_(L)), andcoordinate conversion to the right camera coordinate systemC_(R)=(X_(R), Y_(R), Z_(R)) are given by the following equations:

C _(L) =R _(L) P+T _(L)  (1)

C _(R) =R _(o)(R_(L) P+T _(L))+T _(o)  (2)

where R_(L) and T_(L) are given by the following equations:$R_{L} = {\begin{pmatrix}\cos & {\beta_{L}0} & {{- \sin}\quad \beta_{L}} \\0 & 1 & 0 \\\sin & {\beta_{L}0} & {\cos \quad \beta_{L}}\end{pmatrix}\begin{pmatrix}{\cos \quad \alpha_{L}} & {\sin \quad \alpha_{L}} & 0 \\{{- \sin}\quad \alpha_{L}} & {\cos \quad \alpha_{L}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \quad \gamma_{L}} & {\sin \quad \gamma_{L}} \\0 & {{- \sin}\quad \gamma_{L}} & {\cos \quad \gamma_{L}}\end{pmatrix}}$ T_(L) = (T_(X)^(L), T_(Y)^(L), T_(Z)^(L))^(T)

α_(L), β_(L), and γ_(L) are Euler angles expressing the pose of the leftcamera coordinate system with respect to the reference coordinatesystem, and T_(L) is a translation vector for giving the origin of theleft camera coordinate system. The difference between T_(L) andT_(R)=(T_(X) ^(R), T_(Y) ^(R), T_(Z) ^(R)) which is determined likeT_(L), i.e., the absolute value of T₀=T_(R)−T_(L), gives a base length(B=|T₀|).

When the on-image position of a point obtained by projecting or imaginga point in (X_(L), Y_(L), Z_(L)) onto the sensor of the left camera isgiven by (u^(L), v^(L)) on the left camera coordinate system, (u^(L),v^(L)) is generally given by the following equation: $\begin{matrix}{\begin{pmatrix}u^{L} \\v^{L}\end{pmatrix} = {{\frac{f_{L}}{Z_{L}}\begin{pmatrix}k_{u} & X_{L} \\k_{v} & Y_{L}\end{pmatrix}} + \begin{pmatrix}u_{0}^{L} \\v_{0}^{L}\end{pmatrix}}} & (3)\end{matrix}$

where k_(u) and k_(v) are scaling factors determined by a pixel size.The point (u^(R), v^(R)) corresponding to the right camera is given inthe same manner as described above by using the equation correspondingto the right camera.

Since the feature point coordinates (X₁, Y₁, Z₁), (X₂, Y₂, Z₂), . . .are known, a processing method of estimating the image-pickup parametersby using the measurement results of a plurality of feature points, i.e.,(u₁ ^(L), v₁ ^(L)), (u₂ ^(L), v₂ ^(L)), . . . , and (u₁ ^(R), v₁ ^(R)),(v₂ ^(R), v₂ ^(R)), . . . will be described below. 0₃=(0, 0, 0)^(T) isset, and matrixes M_(L) and M_(R) having unknown image-pickup parametersas elements are given by the following equations:$M_{L} = {\begin{pmatrix}{k_{u}f_{L}} & 0 & 0 & u_{0}^{L} \\0 & {k_{u}f_{L}} & 0 & v_{0}^{L} \\0 & 0 & 1 & 0\end{pmatrix}\begin{pmatrix}R_{L} & T_{L} \\0_{3}^{T} & 1\end{pmatrix}}$ $M_{R} = {\begin{pmatrix}{k_{u}f_{R}} & 0 & 0 & u_{0}^{R} \\0 & {k_{u}f_{R}} & 0 & v_{0}^{R} \\0 & 0 & 1 & 0\end{pmatrix}\begin{pmatrix}{R_{0}R_{L}} & {{R_{0}T_{L}} + T_{0}} \\0_{3}^{T} & 1\end{pmatrix}}$

In the reference coordinate system, the on-image positions U_(a) ^(L=(u)_(a) ^(L), v_(a) ^(L))^(T) and U_(a) ^(R)=(u_(a) ^(R), v_(a) ^(R))^(T)of the left and right cameras at a feature point in A=(X_(a), Y_(a),Z_(a))^(T) can be calculated by the following equations:

W ^(L) =M _(L)(X _(a) Y _(a) Z _(a)1)^(T)  (4)

W ^(R) =M _(R)(X _(a) Y _(a) Z _(a)1)^(T)  (5)

where W^(L)=(s_(L)u_(a) ^(L), s_(L)v_(a) ^(L), s_(L))^(T) andW^(R)=(s_(R)u_(a) ^(R), s_(R)v_(a) ^(R, s) _(R))^(T) and s_(L) and s_(R)are scale parameters which are given by s_(L)=Z_(L) and s_(R)=Z_(RO).

The characteristic feature of the above expression is that a calculationamount is reduced by recasting the estimation of unknown parameters as alinear estimation problem. In this embodiment, a large number of featurepoint data obtained at one image-pickup positionare obtained on thebasis of equations (4) and (5), and image-pickup parameters (cameraposition, pose, base length, vergence angle, focal length, and the like)are estimated by a method of least square or a Kalman filter. Estimationof image-pickup parameters by α_(L), β_(L), γ_(L) and α₀, β₀, γ₀ isreferred to as extraction of left camera reference image-pickupparameters hereinafter. In the present invention, the process ofextracting image-pickup parameters is not limited to the methoddescribed above.

FIG. 4 shows a process flow of a main image measuring operation. Theoutlines of respective processes are described below.

Aberration information measurement process (S1) of the image-pickupsystem is to measure the distortion of a lens. In the present invention,the distortion is measured using a feature pattern 2 having a knownpositional relationship and amethod by a known image processing (e.g.,IEEE journal of Robotics and Automation, vol. RA-3, pp. 323-344, 1987,or IEEE Proceedings of International Conference on Pattern Recognition.1990, pp. 246-253) (in this case, the focal lengths of the left andright cameras may also be measured). In this stage, an image-pickupoperation is not necessarily performed in such a manner that the object3 and the known pattern 2 are set within the same field of view.

FIG. 5 shows the flow of image-pickup parameter measurement process (S2in FIG. 4). In this case, the known pattern 2 and the object 3 arephotographed, and image-pickup parameters (camera position, pose, baselength, vergence angle, focal point position, and the like) areestimated by using a method of least square or a Kalman filter on thebasis of equations (4) and (5). A measurer properly keeps a field ofview while checking an object image with the monitor 5 (e.g., both theobjects are simultaneously set at a predetermined base length and apredetermined vergence angle in the same field range) to photograph theobjects. In this manner, image-pickup parameters are reliably extracted.

First, as initial setting process (S2-0), control of a base length and avergence angle or zooming is performed on the basis of an in-focus statesignal from an optical distance measurement means, arranged on each ofthe left and right cameras 1 _(L) and 1 _(R), for determining anin-focus state, or other rough object distance information, and anobtained image is stored in a image storage means 11 as a still image ora moving image.

A distinguishing process between a known pattern image area and anobject image area in the image is performed (S2-1). In this embodiment,for description, as a known pattern, a colored dot pattern in whichcolored dots (respectively having different hues) arranged at apredetermined distribution density on a planar plate to form a pluralityof concentric circles as shown in FIG. 2A is used. Process (S2-1) isperformed to cutout a colored dot pattern area from the image of theobject 3 placed on the flat plate. In this case, the underlying color orbrightness of the known pattern is set to be different from the averagecolor and brightness of the object 3, and the cutting process isperformed by using a region growing method or the like on the basis ofthe color or brightness level. Although the cutting process is notlimited to this method, the cutting process is performed by using one ofthe structural features, average hue, and average brightness of eachpattern element (dot, in this case) which is different from thecorresponding attribute of the object 3 as a matter of course. Beforethe cutting process, a process of selecting an optimum attribute may beperformed on the basis of the results obtained by measuring theseattributes.

In process step (S2-2), a reference point is set in the known patternarea, and a reference coordinate system for extracting image-pickupparameters and partially three-dimensional shape data is set by usingthe reference point as an origin according to a predetermined method.For example, a coordinate system (X_(i), Y_(i), Z_(i)) at the ithimage-pickup position may be determined such that a Y_(i) axis is set toextend from the origin to the center of the concentric circles, an X_(i)axis is set to be perpendicular to the Y_(i) axis on a plane, and aZ_(i) axis is set to be perpendicular to the X_(i) and Y_(i) axes and tobe a right-hand system. Image-pickup parameters are extracted by using aplurality of feature points in the known pattern image areas in theimages of the left and right cameras. In this case, measurement datafrom the base length adjusting means, the vergence angle adjustingmeans, and focal length adjusting means may be used as initial valuesused when the image-pickup parameters are estimated. Fixed parametersmay be used as fixed ones to estimate other parameters (especially, theposition and pose of the image-pickup means).

In this embodiment, after estimation of image-pickup parameters withreference to the left-camera image is performed, estimation ofimage-pickup parameters with reference to the right-camera image isperformed by using, as initial values, the image-pickup parametersobtained with reference to the left-camera image, or the above processesare repeated until a variation in estimation value becomes a thresholdvalue or less. The processes are to improve accuracy and test theestimation value, and the order of the processes is not limited to aspecific order. In order to stably and reliably perform step (S2-1), theknown pattern area should preferably have features different from thoseon the object 3 (e.g., different in hue, reflectance, spatial frequency,or the like).

In image-pickup parameter recording process (S2-4), image-pickupparameters corresponding to object image data are recorded on arecording medium (not shown) such as a (magneto-optical) magnetic diskor a magnetic tape of the predetermined storage means 12 together withimage data in a predetermined format in which the correspondence ofthese image-pickup parameters is clear.

FIGS. 6 and 7 show recording formats of this embodiment. In FIG. 6,image-pickup parameters of subsequent image data and the address of thecorresponding image data are written in a header section, and left andright image data corresponding to one image-pickup parameter aresequentially written in a data section. In FIG. 7, a header section inwhich image-pickup parameters and image data amount (if necessary) arerecorded immediately before each image data is present. The image datamay be compressed by a proper method. In the header section, associatedattribute information other than the image-pickup parameters, i.e., adate, a measurement site, the name of a measurer, the type of an object,or the like may be recorded. Even if the process steps (S1) and (S2) areperformed in reverse order, or repeated, the effect of this embodimentis not lost.

FIG. 8 shows the flow of three-dimensional shape measurement process (S3in FIG. 4). Process (S3-0) is a process of reading image data andattribute information (including image-pickup parameters) from the imagestorage unit 11. Inter-corresponding-point disparity extraction process(S3-1) of the left and right images of the object 3 from the binocularcamera 1 under the image-pickup conditions as those in process (S2) anddistance information (e.g., a Z-axis coordinate value of a cameracoordinate system) extraction process (S3-2) of each point based on theimage-pickup parameters obtained in process (S2). Although processes(S3-1) and (S3-2) may be performed by using known methods, thereliability of corresponding points is also calculated in process(S3-1).

For example, as the reliability, the following values may be used. Thatis, when block division is performed in process (S3-1), andcorresponding points are extracted on the basis of a correlation valuebetween the left and right image blocks, the correlation value betweenthe blocks may be used as the reliability. A predetermined functionalvalue which is defined on the basis of an assumption of disparitycontinuity and set to be decreased as a degree of discontinuity ofdisparity increases is used as the reliability. In addition, apredetermined functional value (decreases as the corresponding pointcomes closer to an occluding edge) determined depending on the distancebetween the corresponding point and an occluding edge detected by apredetermined method may be used. In this manner, the reliability is seton the assumption that no corresponding point is present near theoccluding edge. Furthermore, the reliability is not limited to the abovevalues, and a compound function constituted by a plurality ofreliability parameters may be used.

In this embodiment, by using a predetermined method (e.g., a featurepoint near the center of the overlapping area between the left and rightimages), a reference point is extracted from the known pattern 2 in theimage obtained at each image-pickup position. A partialthree-dimensional shape of the object 3 is obtained in an orthogonalcoordinate system (the same setting method as that of the referencecoordinate system used in the description in FIGS. 2A and 2B and FIGS.3C and 3D) having the reference point as an origin. In addition, thepartial three-dimensional shape data and, corresponding image-pickupparameters are recorded in a predetermined storage means (medium) in thesame format as described above (S3-3). However, in the presentinvention, the type of the storage means (medium) is not limited to aspecific type.

FIG. 9 shows the flow of three-dimensional shape integration process (S4in FIG. 4). In three-dimensional shape integration process (S4), theplurality of partial three-dimensional shape data of the object 3obtained at different image-pickup positions are connected to each otherat a high accuracy through the reference points set in process (S3) tobe integrated. Since the positions of the reference points and thedirections of the axes on the absolute coordinate system (initially setreference coordinate system) are known, when the three-dimensional shapedata are integrated on the absolute coordinate system, the followingcoordinate conversion process (S4-1) is performed to each data:

P=R _(L)(i)⁻¹(C _(i) −T _(L)(i))  (6)

In the above equation, C_(i), R_(L)(i), and T_(L)(i) representparameters at the ith image-pickup position. These parameters representshapes corresponding to images on the left-camera coordinate system,i.e., the three-dimensional position (vector) at each data point, arotation matrix for the absolute coordinate system of each coordinateaxis, and the position (vector) of an origin, respectively (although theparameters for the right-camera coordinate system can be calculated inthe same manner as described above, the parameters for only one of theleft- and right-camera coordinate systems may be used). Furthermore, asshape errorcorrection process (S4-2), any one of the three-dimensionaldata between overlapping areas between the partial three-dimensionalshape data in integration process is selected on the basis of thereliability calculated in process (S3). For example, when grating pointcoordinates obtained by quantizing (X, Y, Z) coordinate values in apredetermined quantization size coincide with each other, it isconsidered that an overlapping portion is present. In this case, process(S4-2) is applied, and one of the coordinate values before quantizationmay be selected. The overlap determining method is not limited to theabove method as a matter of course. Although either reliability may beused if the reliability satisfies predetermined standards, thereliability having a higher value is basically selected. The shape dataintegrated as described above is recorded on the data storage means 12.

In shape data displaying process (S5 in FIG. 4), the three-dimensionalshape data is read from the shape data storage means 12, display data tobe displayed on the monitor 5 is generated by a method such as a wireframe method. As other displaying methods, images from the left andright cameras may be synthesized with each other to generate a panoramicdisplay, or images from the left and right cameras may bestereoscopically displayed on a head mounted display (binocular display)with disparity distribution.

In this embodiment, since a three-dimensional shape including the bottomsurface of the object 3 that is arranged nearby cannot be extracted byonly placing the object 3 on a planar plate, after the processes up toprocess (S3) are performed within a measurable range, for example, theobject 3 is placed on the known pattern upside down, and the processesfrom process (S*) to process (S3) may be performed. In this case, whenintegrated shape data obtained up to process (S4) are to be finallyintegrated, these shape data especially desirably overlap. As a methodof finally integrating the plurality of integrated shape data into anentire three-dimensional data, the following method may be used. Thatis, with reference to one integrated shape data, the coordinateconversion process (S4-1) is performed to other shape data on theabsolute coordinate system by using various parameter values, a processof determining a degree of overlapping between these shape data isperformed, and coordinate conversion having the highest degree ofoverlapping is employed to select one of the overlapping portions. Thefollowing method may also be used. That is, feature points are extractedin the overlapping area in advance, and the coordinate conversion (S4-1)parameters which realize position alignment of shape data having matchedfeature points are calculated.

Embodiment 2

FIG. 10 is a flow chart of an image-pickup parameter extraction processin Embodiment 2. Respective steps in FIG. 10 correspond to those in FIG.5 of Embodiment 1.

This embodiment has the same arrangement as that of Embodiment 1 exceptthat an image-pickup means is a single-eye camera. In this case, byusing images obtained at ith and (i+1)th (or (i+k)th; k≠1) image-pickuppositions where the images of a known pattern and an object are presentto overlap, an image-pickup parameter extraction process and athree-dimensional shape extraction process based on the image-pickupparameters are performed. In this embodiment, a plurality of featurepoint data are given, and parameters are calculated by using a matrixM_(i) defined by the following equation: $M_{i} = {\begin{pmatrix}{k_{u}f_{i}} & 0 & 0 & u_{0}^{i} \\0 & {k_{u}f_{i}} & 0 & v_{0}^{L} \\0 & 0 & 1 & 0\end{pmatrix}\begin{pmatrix}R_{i} & T_{i} \\0_{3}^{T} & 1\end{pmatrix}}$

as solutions M_(i) and M_(i+m) which are obtained by solving thefollowing equations:

 W ^(i) =M _(i)(X _(a) Y _(a) Z _(a)1)^(T)  (7)

W ^(i+m) =M _(1+m)(X _(a) Y _(a) Z _(a)1)^(T)  (8)

by a method of least square or a Kalman filter.

However, the same repetitive process as in Embodiment 1 is notperformed, and only one parameter may be calculated by the method ofleast square, and the other may be calculated by the Kalman filter. Inthis case, m is an integer number (≠0), W^(i)=(s_(i)u_(a) ^(i),s_(i)v_(a) ^(i), s_(i))^(T), s_(i) is a scale parameter (s_(i)=Z_(i)),and Z_(i) is a Z-coordinate value on a camera coordinate system at theith image-pickup position. A sensor means such as an acceleration sensoror a gyro may be mounted on the image-pickup means to detect a selfmovement parameter, and the parameter may be used to set the initialvalues of position and pose parameters in estimation of image-pickupparameters. As in this embodiment, extraction of image-pickup parametersusing a handy single camera which is smaller than a binocular camera insize, and extraction and integration of three-dimensional shapeinformation of the object based on the image-pickup parameters areperformed.

Embodiment 3

FIGS. 11A and 11B are views for describing a non-planar object having aknown shape, and a feature pattern which is arranged on the surface ofthe object and has features which can be identified and whose positionalrelationship is known.

FIG. 11A shows a case wherein a known feature pattern is formed on aconical table, and FIG. 11B shows a case wherein a known pattern isformed on the inner surface of a bowl-like object. When the knownpattern is arranged on a non-planar surface as in this embodiment, theranges of an image-pickup position and a pose where image-pickupparameters are extracted can be extended without changing a manner ofplacing an object having an unknown shape. In addition, prevention of adegeneracy solution or the like is performed in an image-pickupparameter extraction process (S2 in FIG. 4), thereby realizingimprovements in stability and accuracy.

Embodiment 4

FIG. 12 is a known pattern used in Embodiment 4. Referring to FIG. 12, afeature pattern having features whose positional relationship is knownis formed on a semi-transparent base. This embodiment is characterizedin that the base on which the object 3, having an unknown shape, isplaced can be photographed. In this embodiment, when the feature patternon the semi-transparent base is used, an image-pickup position is placedwithin a range larger than that in Embodiment 3.

In this embodiment, it is desirable that the known pattern is notpresent on a tangent plane of the object 3. The (spectral) reflectioncharacteristics of the front and rear surfaces of the base are desirablymade different from each other, or the attributes of the known patternon the front and rear surfaces are desirably made different from eachother.

The shape of the object 3 near its bottom surface which cannot becovered by the embodiment described above or partial shape data whichcan only be obtained by an image-pickup operation from the lower side ofthe object 3 can be obtained together with the position and pose data ofthe image-pickup means. Therefore, the processes such as the process ofchanging the pose of the object 3 and the processof aligning thepositions of a plurality of integrated data which are performed in theprevious embodiment become unnecessary, and the entire shape of theobject is obtained in fewer calculation steps.

Embodiment 5

FIG. 13 shows the main process flow in Embodiment 5. Image-pickupparameter extraction process (S2 in FIG. 4) in this embodiment isperformed in the same manner as in Embodiment 1 except that a knownpattern is used mainly to extract the position and pose parameters of aimage-pickup means. However, in this embodiment, extraction andintegration of three-dimensional shape data are not performed, andtwo-dimensional images corresponding to the position and pose of theimage-pickup means are recorded on a storage means 12 (S6).

In this case, the process of segmentation/cutting out the known patternand the object image from each other may be performed, and only theobject image may be recorded on the predetermined storage mediumtogether with a viewpoint position. In reproduction, a process (S7) ofcomparing a viewpoint position designated by a user with theimage-pickup position and pose, and an image having, as auxiliaryinformation, image-pickup parameters (in this case, the position andpose data of the image-pickup means) obtained when the parameterscoincides with each other or are very similar to each other is calledfrom the shape data storage means 12 (S8), and the image data isconverted to display data suitable for a display (S5).

For example, a two-dimensional image from one camera may be displayed,images from the left and right cameras are synthesized to perform apanoramic display, or images from the left and right cameras may bestereoscopically displayed on a head mounted display (binocular display)with disparity. In order to smoothly reproduce object images fromarbitrary viewpoint positions with almost the same sizes, animage-pickup operation and a recording operation are desirably performedwhile the distance between the object and the image-pickupmeans is keptas constant as possible. For this purpose, the image-pickup means onwhich no focal length adjusting means is arranged is desirably used,and, especially, in a binocular camera, a base length and a vergenceangle are desirably fixed. However, the following process or the likemust be performed. That is, an optical distance measurement means fordetermining an in-focus state is arranged to control the system so as toprevent an image-pickup operation and parameter extraction from beingperformed in an out-of-focus state.

In Embodiment 5, the position and pose of the image-pickup means arestably extracted at a high accuracy from only an image as describedabove. For this reason, the means according to Embodiment 5 can be usedas a simple image-pickup recording/reproducing means which picks up theimage of an object at various angles to reproduce an object image froman arbitrary viewpoint position designated by a user.

Embodiment 6

FIG. 14 is a view showing the arrangement of Embodiment 6. A projectionmeans 60 in FIG. 14 projects a structure pattern at a predeterminedangle on a non-textured plane and an object 3 placed thereon. As apattern to be projected, a pattern obtained by arranging dots, gratings,or two-dimensional feature pattern elements at a predetermineddistribution density as shown in FIGS. 2A and 2B and FIGS. 3A and 3B maybe used. On the object, the projection pattern has a distributiondensity or a distribution to reflect the shape of the object. Therefore,image-pickup parameters are extracted by using a regularly structuredpatternon the plane from images obtained by performing an image-pickupoperation at image-pickup positions, and a rough three-dimensional shapeis calculated from only the projection pattern on the object surface byusing a known method (“Three-dimensional Image Measurement” by Iguchiand Sato; Shokodo, 1990). In addition, corresponding points (disparity)of the left and right images including the projection pattern areextracted (so-called stereo image measurement) by using a binocularcamera as an image-pickup means, so that a three-dimensional shape datahaving a high density and a high accuracy is obtained.

In this embodiment, in particular, when the rough shape data using aprojection pattern is used as initial shape data in stereo imagemeasurement, shape data of an object whose surface is without texturedpatterns is extracted. For example, the shape data of points other thanfeature points are formed from the initial shape data by aninterpolation process, and the upper or lower limit of correspondingpoints searching range or a disparity value in the stereo imagemeasurement are set by using the shape data, so that three-dimensionalshape data can be extracted at a high speed.

If a pattern is not projected by one projection means depending on animage-pickup position, a plurality of projection means may be arrangedaround the object, and the projection means may be switched to eachother in accordance with an image-pickup position. A table on which theobject 3 is placed need not have a planar surface, and may have a knownshape. However, the surface of the table is desirably non-textured, ifpossible.

Embodiment 7

In Embodiment 7, when the shape of an object 3 is partially known inadvance, a known pattern formed on a predetermined sheet is attached tothe object 3, and the known portion and a portion having an unknownshape are photographed, within the same field of view. Image-pickupparameters are extracted from the observed feature points. In addition,a known three-dimensional shape portion is extracted on the basis of theimage-pickup parameters in the same manner as in Embodiment 1.

When partial shapes are to be obtained by changing image-pickuppositions, a pattern is attached to a portion having a known shape datapresent in an image obtained at the image-pickup position, and the sameprocesses as described above are repeated. In this embodiment, theobject 3 is desirably constituted by a smooth surface, i.e., a planarsurface or a curved surface having a small curvature.

According to Embodiment 7, the three-dimensional shape of an object issimply extracted although the object has a relatively large size.

Embodiment 8

In Embodiment 8, a binocular image-pickup means (not shown) includes twoeyes respectively having different fixed focal lengths. A maincontroller 10 controls the image-pickup means such that one eye is usedto acquire a known pattern image and the other eye is used to acquirethe image of an object having an unknown shape.

The remaining arrangement of Embodiment 8 is the same as that ofEmbodiment 1. In this manner, an error of image-pickup parameters orthree-dimensional shape data caused by one blurred image obtained whenan unknown-shape object and a known pattern are photographed within thesame field of view is suppressed. In this embodiment, the sameimage-pickup parameter extraction process as that in Embodiment 2 isperformed.

What is claimed is:
 1. An image processing method characterized in thatsensing an object with a predetermined image pattern whose shape andposition are known from a plurality of viewpoint positions, extractingimage data corresponding to the predetermined image pattern andoperating parameters concerning image sensing positions and imagesensing directions by detecting a change of the predetermined imagepattern in the image data, and producing a three-dimensional image ofthe object sensed from an arbitrary viewpoint by using the image of theobject sensed from the plurality of viewpoint positions and theparameters concerning image sensing position and image sensingdirection.
 2. An image processing method according to claim 1,characterized in that the predetermined image pattern is present on anobject whose shape is at least partially known.
 3. An image processingmethod according to claim 1, wherein the known pattern is constituted bya plurality of patterns respectively having features which are differentfrom each other.
 4. An image processing apparatus according to claim 1,wherein the known pattern is aligned such that attributes of features ofa shape and an image change according to a predetermined rule.
 5. Animage processing method characterized in that sensing a predeterminedimage pattern having features which are known is defined as a firstobject, and a predetermined object is defined as a second object, andoutputting an image comprising both of a first image and a second imagecorresponding to the first object and the second object, respectively,and extracting parameters concerning image sensing positions and imagesensing directions of image sensing means by extracting the first imagefrom the image comprising both the first and second images and detectinga change of the first images in the first images of the first objectfrom a plurality of viewpoint positions, and producing athree-dimensional model image of the second object sensed from anarbitrary view position by using the second images sensed from theplurality of viewpoint positions and the parameters concerning imagesensing positions and image sensing directions of image sensing meanscorresponding to the second images.
 6. An image processing apparatuscomprising: input means for inputting images respectively sensed both ofa first object image whose shape and position are known and a secondobject image having a three-dimensional shape which is at leastpartially unknown, and each of which is obtained by viewing at aplurality of viewpoint positions; parameter extracting means foroperating parameters concerning image sensing positions and sensingdirections by extracting the first object image from the image input bysaid input means and detecting a change of the first object image;three-dimensional shape information extracting means for extractingthree-dimensional shape information of the second object on a basis ofthe parameters; recording means for recording the second object imagesat a plurality of viewpoint positions and the three-dimensional shapeinformation corresponding to the second object images respectively; andimage display means for displaying a three-dimensional image viewed fromone of said arbitrary positions by synthesizing the plurality of thesecond object images on the basis of the three-dimensional shapeinformation and one of said arbitrary positions.
 7. An image processingapparatus according to claim 6, wherein said input means includesimage-pickup means for photographing an object and storage means forstoring an image.
 8. An image processing apparatus according to claim 7,wherein said image-pickup means has a plurality of optical paths whosefixed focal lengths are different from each other, one optical path isused to acquire the first object image, and the other optical path isused to acquire the second object image.
 9. An image processingapparatus according to claim 6, wherein said three-dimensional shapeinformation extracting means includes a reference coordinate systemsetting means having a predetermined point on the first object as anorigin and a three-dimensional shape information extracting means forthe second object in the reference coordinate system.
 10. An imageprocessing apparatus according to claim 6, wherein the first object isconstituted by arranging light-emitting elements in a predeterminedpattern.
 11. An image processing apparatus according to claim 4, whereinthe features of the image are a continuous change in hue, picturepattern, and image pattern.
 12. An image processing apparatus accordingto claim 11, wherein the known pattern has portions which are differentfrom each other in the hue, picture pattern, and image pattern.
 13. Animage processing apparatus according to claim 4, wherein the pattern ison a conic section or a curve expressed by an n-degree polynomial. 14.An image processing apparatus according to claim 4, wherein a directionof the pattern is in circumferential and radial directions of an ellipseor a circle.
 15. An image processing apparatus according to claim 4,wherein the pattern is an n-angle polygon.
 16. An image processingapparatus according to claim 4, wherein the rule changes depend on adirection of a pattern wherein the known pattern creeps.
 17. An imageprocessing apparatus according to claim 16, wherein, when the pattern isa concentric pattern, attributes change according to different rules inan R (radial) direction and a theta direction, respectively.
 18. Animage processing apparatus comprising: input means for inputting imagesrespectively sensed both of a first object image whose shape andposition are known and a predetermined second object, obtained byviewing, at a plurality of viewpoint positions; parameter extractingmeans for operating parameters concerning image sensing positions andsensing directions by extracting from the image input, by said inputmeans, the first object image and detecting a change of the first objectimage; image recording means for recording the second object images at aplurality of viewpoint positions; parameter recording means forrecording parameters concerning the image sensing positions and sensingdirections corresponding to the second object images respectively; andimage display means for displaying a three-dimensional image viewed froman arbitrary position by synthesizing the plurality of object images onthe basis of the parameters and the arbitrary position.
 19. An imageprocessing method comprising: a step of setting a predetermined point onan image pattern whose shape and position are known as a referencepoint; a step of setting a predetermined coordinate having the referencepoint as a center; a step of simultaneously sensing an object set on thepredetermined image pattern and the image pattern; a step of extractinga partial three-dimensional shape of the object corresponding to thereference point by using images sensed from a plurality of viewpointpositions to include the reference point; a step of obtaining acoordinate conversion parameter for performing a conversion of thepredetermined coordinate between different reference points by detectinga change of the predetermined image pattern; and a step of producing athree-dimensional model image at an arbitrary viewpoint position byintegrating a plurality of partial three-dimensional shapes of theobject obtained at the different reference points on a basis of thecoordinate conversion parameter.
 20. An image processing methodaccording to claim 19, wherein, in a process of extracting parameters byusing the plurality of images obtained from the plurality of viewpointpositions, as the parameters, the parameters being extracted withreference to the other viewpoint position by using parameters extractedwith reference to one viewpoint position to estimate reliability of theparameters obtained with reference to the viewpoint positions.
 21. Animage processing method according to claim 20, wherein thethree-dimensional shape of an object whose shape is at least partiallyunknown by using the parameters obtained by estimating reliability. 22.An image processing method according to claim 19, wherein thethree-dimensional extraction process records parameters duringphotographing of the object, and reads the parameters during extractionof the three-dimensional shape to extract the three-dimensional shape.23. An image measuring method according to claim 19, wherein thethree-dimensional shape extraction process includes a step of recordingthe parameters and object images in a predetermined format duringphotographing of the object, and reading an object image having aparameter which is closest to a view point position designated duringreproduction of the object images to reproduce the object image.